Search for: All records

Integrating multimodal data such as RGB and LiDAR from multiple views significantly increases computational and communication demands, which can be challenging for resource-constrained autonomous agents while meeting the time-critical deadlines required for various mission-critical applications. To address this challenge, we propose CoOpTex, a collaborative task execution framework designed for cooperative perception in distributed autonomous systems (DAS). CoOpTex contribution is twofold: (a) CoOpTex fuses multiview RGB images to create a panoramic camera view for 2D object detection and utilizes 360° LiDAR for 3D object detection, improving accuracy with a lightweight Graph Neural Network (GNN) that integrates object coordinates from both perspectives, (b) To optimize task execution and meet the deadline, CoOpTex dynamically offloads computationally intensive image stitching tasks to auxiliary devices when available and adjusts frame capture rates for RGB frames based on device mobility and processing capabilities. We implement CoOpTex in real-time on static and mobile heterogeneous autonomous agents, which helps to significantly reduce deadline violations by 100% while improving frame rates for 2D detection by 2.2 times in stationary and 2 times in mobile conditions, demonstrating its effectiveness in enabling real-time cooperative perception. 

Robust communication is vital for multi-agent robotic systems involving heterogeneous agents like Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs) operating in dynamic and contested environments. These agents often communicate to collaboratively execute critical tasks for perception awareness and are faced with different communication challenges: (a) The disparity in velocity between these agents results in rapidly changing distances, in turn affecting the physical channel parameters such as Received Signal Strength Indicator (RSSI), data rate (applicable for certain networks) and most importantly "reliable data transfer", (b) As these devices work in outdoor and network-deprived environments, they tend to use proprietary network technologies with low frequencies to communicate long range, which tremendously reduces the available bandwidth. This poses a challenge when sending large amounts of data for time-critical applications. To mitigate the above challenges, we propose DACC-Comm, an adaptive flow control and compression sensing framework to dynamically adjust the receiver window size and selectively sample the image pixels based on various network parameters such as latency, data rate, RSSI, and physiological factors such as the variation in movement speed between devices. DACC-Comm employs state-of-the-art DNN (TABNET) to optimize the payload and reduce the retransmissions in the network, in turn maintaining low latency. The multi-head transformer-based prediction model takes the network parameters and physiological factors as input and outputs (a) an optimal receiver window size for TCP, determining how many bytes can be sent without the sender waiting for an acknowledgment (ACK) from the receiver, (b) a compression ratio to sample a subset of pixels from an image. We propose a novel sampling strategy to select the image pixels, which are then encoded using a feature extractor. To optimize the amount of data sent across the network, the extracted feature is further quantized to INT8 with the help of post-training quantization. We evaluate DACC-Comm on an experimental testbed comprising Jackal and ROSMaster2 UGV devices that communicate image features using a proprietary radio (Doodle) in 915-MHz frequency. We demonstrate that DACC-Comm improves the retransmission rate by ≈17% and reduces the overall latency by ≈12%. The novel compression sensing strategy reduces the overall payload by ≈56%.